It is common for product configurations presented on seller websites to be periodically refreshed with newer or different components, attributes, prices, and buyer-selectable options. In some cases, listed products and/or such configurations may expire and/or be replaced with entirely different products or product features.
In many data processing systems for managing such web sites, data representing the products is stored in upstream systems (e.g., web site servers) which process and query backend systems for periodic updates. There could be millions or more changes that occur across a group of products over a refresh cycle and the amount of data needing to be processed in the upstream systems can subsequently result in heavy bottlenecks.
Expirations of various products and sub-components also typically need to be handled by the upstream data processors as they survey the downstream processors (e.g., product vendors) for updates during each refresh cycle. Additionally, the changes that happen to sub components of the products can affect hundreds of thousands or more products across multiple regions and catalogs, which typically require heavy reliance on relatively few upstream data processors.
Thus, new techniques for improving the efficiency of such data processing systems and eliminating the aforementioned bottlenecks is desired.